1. Field of the Invention
Embodiments of the present invention relate to image processing. More particularly, embodiments of the present invention relate to systems and methods for image processing that subtract successive images to provide a resultant composite image, for purposes such as ambient light canceling, motion clutter suppression, or motion detection.
2. Background Information
Within the field of photography and image processing, a class of operations involves the subtraction of successive images captured from a camera to obtain a desired effect in the resultant composite image. For example, image subtraction is used to capture and emphasize lighting differences arising from alternate forms of scene illuminations, or to detect moving objects within a scene.
One example of image subtraction is ambient-light-canceling (ALC) photography. A camera captures two successive images of a scene, one with a camera lamp illuminating the scene and the other with the camera lamp off, and the two images are subtracted to form a synthetic image. Assuming that neither the camera nor the scene has moved between the two image-capture periods, and that the ambient light, i.e., the light coming from sources other than the camera lamp, has remained relatively constant between the two image captures, the subtracted image represents what the scene would look like if it were illuminated by the camera lamp only, without the ambient light. The subtraction of the two successive images cancels the effect of the ambient light, but leaves the effect of the camera lamp.
The process of eliminating ambient light from a camera image by subtracting two successive images, one image taken with the camera illuminator on and the other camera image taken with the camera illuminator off, is described in U.S. Pat. No. 4,274,735 by Tamura et al., U.S. Pat. No. 4,315,159 by Niwa et al., U.S. Pat. No. 4,490,037 by Anagnostopoulos et al., U.S. Pat. No. 5,287,183 by Thomas et al., U.S. Pat. No. 6,021,210 by Camus et al., and U.S. Pat. No. 6,256,067 by Yamada.
Image subtraction, however, is highly sensitive to scene motion, either to motion of the camera or to motion of objects within the scene. Indeed, because image-subtraction is sensitive to image motion, motion detection is a second key application of image subtraction. If there is motion between the successive images, all or portions of the two images become spatially displaced with respect to each other, and the image subtraction results in residuals around the edges of the moving objects.
While it is desired to emphasize the effects of motion in motion-detection applications, it is often desired to minimize motion effects in other applications of image subtraction. In applications where motion effects of image-subtraction are undesirable, these motion effects are often called ‘motion clutter.’ Motion clutter, for example, is generally highly undesirable in ambient-light-cancellation applications.
While the term “camera” typically implies the generation of images from light, one skilled in the art will appreciate that the term “camera” is contemplated to include all forms of imagers that generate images from any form of physical wave, including vibrational waves and/or particle wavefronts. Waves include, for example, visible and non-visible light, electromagnetic, radar, x-rays, gamma rays, electron, magnetic, magnetic-resonance, sonic, ultrasonic, seismic, surface, body, compression, and longitudinal waves. Particle wavefronts include nuclear particles (such as electrons, protons, positrons, neutrinos, etc.) atomic particles, molecular particles, and other material particles. The concepts discussed here apply to all wavelengths, visible or invisible, audible or inaudible. Additionally, the concepts apply to both single-frame and video modes of camera operation. Finally, though images are typically considered to be two-dimensional, the concepts discussed here apply to any number of image dimensions, including 1, 2, 3, and more.
Ambient-light-cancelling Photography
Camera images formed with multiple light sources on a scene are conceptually equivalent to the superposition of multiple images, each formed with individual light sources. Once an aggregate image is formed however, it is generally not possible to separate its constituent images. If a specialized camera properly captures an appropriate set of constituent images ahead of time, however, the desired synthetic images can be generated computationally, typically by an additive or subtractive operation on the constituent images on pixel-by-pixel basis.
It is known that the effects of ambient light may be largely reduced or eliminated from a camera's image by taking two consecutive images, one with the camera's controlled illuminator on and the other with the camera illuminator off, and then subtracting the two images to form the composite ambient-light-cancelled image.
Assuming that the ambient illumination remains approximately constant during the two image periods, the subtraction of the off-cycle image from the on-cycle image results in a net-zero contribution from the ambient light source, while leaving the full result of the camera's illumination in the resultant image.
FIGS. 1-3 illustrate the results of ambient light cancellation achieved using image subtraction. The objective of this camera is to detect (i.e. highlight) objects in the foreground and ignore (i.e. suppress) objects in the background. In this example scene, there is a person in the foreground, there is a house and a parked car in the near background, and there is a (unrealistically) stationary airplane in the far background. The ambient illumination is normal daylight. The camera's controlled illuminator is a flash attachment.
The first camera image A, 100 shown in FIG. 1, is taken with a flash attachment on the camera. The second camera image B, 200 shown in FIG. 2, is taken without a flash. Due to the daytime condition, the foreground and background objects are illuminated equally by the ambient daylight. The flash is not particularly bright with respect to the ambient daylight, so the person in the foreground is only slightly highlighted in FIG. 1 in comparison to FIG. 2.
The final output image C, 300 shown in FIG. 3, represents the “subtracted” or “difference” image, i.e. the image 300 in FIG. 3 equals the image 100 in FIG. 1 minus the image 200 in FIG. 2. (The difference image C, 300 in FIG. 3 is also magnified for purposes of displaying the contrast in the resultant composite image.) The key results of the subtracted image C, 300 in FIG. 3 are that the objects in the foreground are highlighted, and the objects in the background are suppressed.
The person in the foreground now stands out significantly from the background, making objects in the foreground far easier to detect with respect to objects in the background. Note that the subtracted image C, 300 in FIG. 3 is the same as if the picture were taken with a flash at night, without ambient daylight; the illumination from the flash highlights objects in the foreground.
Motion Detectors
Image-subtraction photography is also used to detect objects that are moving within a scene. When using image subtraction to perform motion detection, typically both the first and second images are illuminated equally. When the two images are subtracted, all the stationary objects are ‘cancelled out’ in the composite image, leaving only a uniform “neutral-gray” response. If an object moves, however, the images of the object are spatially separated between the first and second constituent images, so the subtraction does not result in pure cancellation of the object image; rather, the subtraction results in a residual ‘ghost’ image around the edges of the moving object. The existence of the ghosted image within the otherwise neutral-gray background allows the moving object to be detected easily within the subtracted image.
As an object moves, the ghosting effect occurs around the leading and trailing edges of the object. Edges of an object are typically characterized by a change, i.e., a gradient, in the image intensity. As an object moves between two successive images, the differences in the leading and trailing edge locations between the two successive images produce a difference in intensity that shows up as non-neutral-gray in the difference image. It is the emergence of the edges of the moving objects that produces the ghosting effect.
Since the ghosting appears along the leading and trailing edges of moving objects, the ghost image emphasizes the object's direction of motion. If the surrounding background of an object is brighter than the object, the leading edge of the ghost image appears brighter than neutral gray. In the first image, the area just ahead of the moving object is the brighter background intensity. In the second image, that area is replaced by the darker intensity of the object that has moved into the space. When the two images are subtracted, the lower (darker) intensity of the second image is subtracted from the brighter (higher) intensity of the first image, resulting in a net brighter-than-neutral-gray in the difference image.
Conversely, if the background is darker than the object, the leading edge of the object appears darker than neutral gray. Similarly, the trailing edges of dark objects are dark, and the trailing edges of bright objects are bright.
FIGS. 4-6 illustrate the results of motion detection achieved using subtraction of successive images. In this example, the scene is the same as in the above ambient-light-cancellation example shown in FIGS. 1-3, except that the car and plane are now in motion. The illumination is identical for both of the constituent photos. The two successive raw (i.e. constituent) images A, 400 and B, 500 are shown in FIGS. 4 and 5, respectively. The difference image C, 600, i.e., the image A, 400 in FIG. 4 minus the image B, 500 in FIG. 5, is shown in FIG. 6. The key results of the subtracted image are that the moving objects have been highlighted, and the stationary objects have been suppressed.
Note the ghosting effect on the highlighted moving objects, i.e., the ghosting on the car and the plane. Since the surrounding backgrounds of both the car and plane are brighter than the vehicles themselves, the leading edges of these objects are bright and the trailing edges are dark. It can thus be deduced from the ghost images that both vehicles are moving leftward.
Also note that in image-subtraction, the subtraction process results in positive intensities in areas where the ‘added’ image is brighter than the ‘subtracted’ image, and results in negative intensities where the ‘subtracted’ image is brighter than the ‘added’ image. To accommodate negative intensities when displaying difference images, the difference image is customarily drawn with a “neutral gray” value so that all display intensities are positive and can be interpreted visually. In areas of the two constituent images where both intensities are equal, the subtracted output takes the value of the neutral-gray offset. Brighter-than-neutral-gray regions indicate that the ‘added’ image is brighter, and darker-than-neutral-gray regions indicate that the ‘subtracted’ image is brighter.
Image Subtraction Timing
FIG. 7 illustrates the timing for a typical image-subtraction frame capture. Periods A, 71, and B, 72, represent the shutter periods during which two constituent images A and B are captured. The durations of the two periods are ΔtA and ΔtB, and the overall frame period for the composite image, 73, includes both ΔtA and ΔtB. To achieve the desired image-subtraction effects, the lighting conditions for the two images are typically synchronized to their respective shutter periods.
As shown in FIG. 7, conventional image subtraction involves the subtraction of two images, where the full-scaled intensity profile of one captured image is subtracted from the full-scale intensity profile of the second to form the final image.
Motion Clutter
In many image-subtraction applications, such as ambient light cancellation, the ghosting effects of object motion are undesired. When the ghosting is unwanted, it is referred to as “motion clutter.” Physically, the desired ghosting effects in motion detectors and the undesired motion clutter effects in ambient-light-cancellation applications are identical. The difference lies only in the desirability of the effect when images are subtracted.
Motion clutter, like ghosting, is measured in units of image brightness, i.e., image intensity. In motion detection applications, the reference level for “zero ghosting” is neutral gray. At all positions in the image where there is no moving object, the subtracted image is neutral gray, and therefore no ghosting exists. As the intensity of the subtracted image moves away from neutral gray, either positively or negatively, the magnitude of the ghosting increases. In ambient-light-cancellation applications, the reference level for “zero motion clutter” is black. The magnitude of motion clutter is measured as an increase in brightness from black.
Note that motion clutter is a different phenomenon than motion blur, though both result from motion in the camera scene. Blur results from scene motion during the overall frame period, whether or not image subtraction is performed. Motion blur can be seen on the moving car in FIGS. 4 and 5, where subtraction has not yet been performed. Motion clutter, i.e., the ghosting phenomenon illustrated in FIG. 6, on the other hand, only results when images are subtracted, as it is in motion-detection and ambient-light-cancellation photography, to form the final image.
In image subtraction, the spatial displacement Δx of a moving object in two successive camera images is proportional to both the velocity V of the objects in the scene and the average temporal offset Δt between the shutter periods of the two images:Δx=V×Δt  (Eq 1)
This quantity Δx, i.e., the average spatial displacement between images, is also called the ‘motion clutter width,’ which refers to the width of the ghosting edges that appear in the motion clutter. It is assumed in the above equation that there is no appreciable “dead time” between the two image shutter periods. Dead time between shutter periods further increases image motion clutter.
If the shutter periods ΔtA and ΔtB are different, the average time difference Δt between two successive shutter periods is equal to the average of ΔtA and ΔtB:Δt=(ΔtA+ΔtB)/2  (Eq 2)
An example of undesired motion clutter is illustrated in FIGS. 8-10. In this example, the camera's objective is to highlight objects in the foreground. Highlighting of foreground objects is accomplished by ambient light cancellation, i.e., by image subtraction and using a controlled flash during one of the image captures. The camera configuration is the same as that used to generate the images in FIGS. 1-3 (ambient light cancellation). In this new example, however, the car and the plane in the background are now moving, while they were still in the example of FIGS. 1-3. Note that the ghosted images of the moving car and plane now appear in the composite, subtracted image, shown in FIG. 10. Furthermore, the ghosted images of the car's tires are even brighter than the person in the foreground. In this example, where the camera's objective is to highlight objects in the foreground and suppress images in the background, the ghosted images of the moving car and plane in the background are undesirable.